Another alternative is to not only measure one or both of the above, but to also analyze the impact of other factors, so that we can not only identify the outputs of the program, but break down those outputs in terms of changes in inputs. For example, next year we will be tracking the race of the coaches. This can help us to determine if matching coaches and students by race has any impact on the students' success rates. More in-depth analysis will require more money to measure. Also, it may not provide useful data unless the pool of information is larger, which is ultimately a function of available funding. The benefit, however, is that this information will allow us to increase the effectiveness of the program, so as to get better use out of the money we spend.
Evaluation Criteria
The most important evaluation criteria is the specificity of the information. There are so many variables in education that it can be difficult to accurately gauge the effectiveness of a program like Team Read. We need to seek out the information that not only provides specific and direct feedback but isolates as many key variables as possible.
The second most important criteria is that whatever method we choose should address the needs of the most important stakeholder - the readers. Their educational needs should ultimately guide the structure of Team Read, and we should be able to gather information that helps us to understand those needs and the best ways to address them.
The third criteria should be timeliness. There is little doubt that Team Read will proceed next year, but there is doubt as to the specific level of funding that will be made available. We need to be able to provide feedback to the financiers as soon as possible so that we can adequately prepare for the coming school year, in terms of having a budget and being able to identify the number of schools we are able to help. The ad hoc manner in which we grew this year was a necessary requirement of a first-year program, but as the size of the program increases both those of us administering Team Read and those funding it will need to have greater certainty. Every summer we will have two months to wrap up one school year and prepare for the next. The approximate timeline...
This means that we have a short window in which to gather information on the effectiveness of the program.
Cost is not considered to be an issue, since most of the data will be available fairly easily and inexpensively, certainly relative to the cost of the program. The government provides us with test results, and we will track most of the other information ourselves.
Conclusions
In order to best meet the needs of the readers, we wish to gather as much information as possible. The primary information we propose to use to measure the effectiveness of the program are the year-over-year results for the students involved in the program. This will be weighed against the results for students of equivalent performance from the same schools but who are not in the program. By comparing classmates we hope to eliminate many of the variables that can make measurement difficult. We feel that school test scores are subject to too many of these variables in order to provide significant value as a tool to measure the effectiveness of a program. After all, less than 1% of students are in the program, which means their results will be drowned out if we look at entire student bodies.
We also wish to build a better program, and this means we would like to gather more information about both the students and the coaches. If there are any trends with regards to the effectiveness of specific pairings, hopefully we can use that to tailor a more effective program in the future. This will add to the time to process the statistics and will require a greater degree of primary information gathering on our part, but will provide an extra measure of security for the financiers that the program will be as effective as possible.
By approaching effectiveness measurement in this way we hope to break down how effective the program is for the financiers, find ways to improve the results for the readers, and to gather the information quickly enough to set budgets and operational strategies for the coming school year in the two months we have during the summer months.
